Users typically engage with LLMs interactively, yet most existing benchmarks evaluate them in a static, single-turn format, posing reliability concerns in interactive scenarios. We identify a key obstacle towards reliability: LLMs are trained to answer any question, even with incomplete context or insufficient knowledge. In this paper, we propose to change the static paradigm to an interactive one, develop systems that proactively ask questions to gather more information and respond reliably, and introduce an benchmark--MEDIQ--to evaluate question-asking ability in LLMs. MEDIQ simulates clinical interactions consisting of a Patient System and an adaptive Expert System; with potentially incomplete initial information, the Expert refrains from making diagnostic decisions when unconfident, and instead elicits missing details via follow-up questions. We provide a pipeline to convert single-turn medical benchmarks into an interactive format. Our results show that directly prompting state-of-the-art LLMs to ask questions degrades performance, indicating that adapting LLMs to proactive information-seeking settings is nontrivial. We experiment with abstention strategies to better estimate model confidence and decide when to ask questions, improving diagnostic accuracy by 22.3%; however, performance still lags compared to an (unrealistic in practice) upper bound with complete information upfront. Further analyses show improved interactive performance with filtering irrelevant contexts and reformatting conversations. Overall, we introduce a novel problem towards LLM reliability, an interactive MEDIQ benchmark and a novel question-asking system, and highlight directions to extend LLMs' information-seeking abilities in critical domains.

Users typically engage with LLMs interactively, yet most existing benchmarks evaluate them in a static, single-turn format, posing reliability concerns in interactive scenarios. We identify a key obstacle towards reliability: LLMs are trained to answer any question, even with incomplete context or insufficient knowledge. In this paper, we propose to change the static paradigm to an interactive one, develop systems that proactively ask questions to gather more information and respond reliably, and introduce an benchmark--MEDIQ--to evaluate question-asking ability in LLMs. MEDIQ simulates clinical interactions consisting of a Patient System and an adaptive Expert System; with potentially incomplete initial information, the Expert refrains from making diagnostic decisions when unconfident, and instead elicits missing details via follow-up questions. We provide a pipeline to convert single-turn medical benchmarks into an interactive format.

I am a forty-five-year-old journalist who, for many years, didn't read the news. In high school, I knew about events like the O. J. Simpson trial and the Oklahoma City bombing, but not much else. In college, I was friends with geeky economics majors who read The Economist, but I'm pretty sure I never actually turned on CNN or bought a paper at the newsstand. I read novels, and magazines like Wired and Spin. If I went online, it wasn't to check the front page of the Times but to browse record reviews from College Music Journal. Somehow, during this time, I thought of myself as well informed.

This paper introduces AskSport, a question-answering web application about sports. It allows users to ask questions using natural language and retrieve the three most relevant answers, including related information and documents. The paper describes the characteristics and functionalities of the application, including use cases demonstrating its ability to return names and numerical values. AskSport and its implementation are available for public access on HuggingFace.

In recent years, large language models (LLMs) have shown an impressive ability to perform arithmetic and symbolic reasoning tasks. However, we found that LLMs (e.g., ChatGPT) cannot perform well on reasoning that requires multiple rounds of dialogue, especially when solving situation puzzles. Specifically, LLMs intend to ask very detailed questions focusing on a specific aspect or same/similar questions after several rounds of Q&As. To help LLMs get out of the above dilemma, we propose a novel external reformulation methodology, where the situation puzzle will be reformulated after several rounds of Q&A or when the LLMs raise an incorrect guess. Experiments show superior performance (e.g., win rate, number of question/guess attempts) of our method than directly using LLMs for solving situation puzzles, highlighting the potential of strategic problem reformulation to enhance the reasoning capabilities of LLMs in complex interactive scenarios.

During CES 2025, I had a chance to check out a demo of the way Google is integrating Gemini capabilities into its smart home platform via devices like the Nest Audio, Nest Hub and Nest Cameras. The main takeaway is that the conversations you have with the Google Assistant will feel more natural. Personally, I'd appreciate being able to ask questions as they pop in my head, without having to formulate some Assistant-friendly sentence before speaking -- what I saw makes me feel like my wish could come true. To kick things off, you'll still say "Hey Google," but for follow-up questions you can skip the prompt and the Assistant will be able to hold on to the thread of your conversation. During the demonstration, held in a simulated (and very posh) kitchen, the Google representative asked things like what to cook with ingredients he had on hand (chicken and spinach).

Chat GPT is not the only AI service in town. Google Gemini is a similar service where you can ask questions and get answers in plain text–no commands required. You can "converse" just as if the AI robot were a real person. If you're familiar with Chat GPT, you'll recognize it because the layout is similar. You're greeted by a stripped-down screen with a text input field at the bottom.

Artificial intelligence--and in particular generative AI--continues to push its way into every aspect of digital life, with varying degrees of success. One of the latest updates from Google adds the Gemini AI chatbot to Gmail on Android and iOS, which means you can ask questions about anything in your inbox. For example, you might want a summary of a discussion you've been having with your boss or need a reminder about when an upcoming camping trip is actually happening. For queries like these, Gemini can dive into your email threads and pull out the salient details for you. This is separate to the Gemini text creation tools you get when composing emails in Gmail on the web, and--for the time being at least--it's exclusive to those with a paid Google Workspace account or a subscription to the Google One AI Premium plan.

You're neck deep in a research project but the finish line is in sight. You hit the close button on your browser. It vanishes and takes the dozens of tabs you had open with it. You heave a sigh of relief -- and then remember that you need to verify just one more detail from one of the web pages you had open. The problem is that you have no idea which one it was or how to get back there.

Interactive Artificial Intelligent(AI) assistant systems are designed to offer timely guidance to help human users to complete a variety tasks. One of the remaining challenges is to understand user's mental states during the task for more personalized guidance. In this work, we analyze users' mental states during task executions and investigate the capabilities and challenges for large language models to interpret user profiles for more personalized user guidance. In the digital age, there is immense potential for artificial intelligent (AI) assistant to guides users through complex tasks, from changing laptop batteries to piping frosting on a cake. One of the main challenges, however, lies in creating an interactive system that can not only understand which step the user is at, but can also detect user's mental states, such as frustration, familiarity with the task, detail-orientation, etc.